Comparing Adaptive and Non - AdaptiveConnection Pruning With Pure Early

|Neural network pruning methods on the level of individual network parameters (e.g. connection weights) can improve generalization, as is shown in this empirical study. However, an open problem in the pruning methods known today (OBD, OBS, autoprune, epsiprune) is the selection of the number of parameters to be removed in each pruning step (pruning strength). This work presents a pruning method lprune that automatically adapts the pruning strength to the evolution of weights and loss of generalization during training. The method requires no algorithm parameter adjustment by the user. Results of statistical signiicance tests comparing autoprune, lprune, and static networks with early stopping are given, based on extensive experimentation with 14 diierent problems. The results indicate that training with pruning is often signiicantly better and rarely signiicantly worse than training with early stopping without pruning. Furthermore, lprune is often superior to autoprune (which is superior to OBD) on diagnosis tasks unless severe pruning early in the training process is required. The principal idea of pruning is to reduce the number of free parameters in the network by removing dispensable ones. Pruning methods usually either remove complete input or hidden nodes along with all their associated parameters or remove individual connections, each of which carries one free parameter (the weight). This latter approach is very ne-grained and makes pruning particularly powerful. If applied properly, pruning often reduces overrtting and improves generalization. At the same time it produces a smaller network. Interestingly, most papers on pruning algorithms do show empirically that smaller networks can be obtained without loss of generalization, but do not show that generalization will often be improved compared to reasonable static-network training methods. The present paper makes up for that. The key to pruning is a method to calculate the approximate importance of each parameter. Several such methods have been suggested. The simplest one | with obvious aws 3] | is to assume the importance to be proportional to the magnitude of a weight. More sophisticated approaches are the well-known optimal brain damage (OBD) and optimal brain surgeon (OBS) methods. OBD 1] uses an approximation to the second derivative of the error with respect to each weight to determine the saliency of the removal of that weight. Low saliency means low importance of a weight. OBS 5] avoids the drawbacks of the approximation by computing the second derivatives (almost) exactly, but is computationally very expensive. Both methods have the …